Robust sex differences in some spatial abilities that favor males have raised the question of whether testosterone contributes to those differences. There is some evidence for prenatal organizational effects of testosterone on male-favoring spatial abilities, but not much is known about the role of pubertal testosterone levels on adult cognitive abilities. We studied the association between pubertal testosterone (at age 14) and cognitive performance in young adulthood (at age 21–23), assessing male-favoring, female-favoring, and sex-neutral cognitive domains in a population-based sample of 130 male and 178 female twins. Pubertal testosterone was negatively associated with performance in the Mental Rotation Test in young adult men (r = −.27), while among women no significant associations between testosterone and cognitive measures were detected. The significant association among men remained after controlling for pubertal development. Confirmatory within-family comparisons with one-sided significance testing yielded a negative correlation between twin pair differences in testosterone levels and Mental Rotation Test performances in 35 male twin pairs (r = −.32):the twin brother with higher testosterone performed less well on the Mental Rotation Test. That association was evident in18 pairs of dizygotic male twin pairs (r = −.42; analysis controlling for shared environmental effects). In contrast, the association of differences was not evident among 17 monozygotic male twin pairs (r = −.07; analysis controlling for shared genetic influences). Results suggest that pubertal testosterone levels are related specifically to male-favoring spatial ability and only among men. Within-family analyses implicated possible shared genetic effects between pubertal testosterone and mental rotation ability.